Skip to content

Improve admin mapping change output#1206

Open
jwils wants to merge 2 commits into
mainfrom
joshuaw/hide-delete-changes-admin-lambda
Open

Improve admin mapping change output#1206
jwils wants to merge 2 commits into
mainfrom
joshuaw/hide-delete-changes-admin-lambda

Conversation

@jwils

@jwils jwils commented May 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Move mapping-removal noise handling into elasticgraph-admin so all admin users benefit.
  • Preserve current mapping fields in concrete index update payloads and diffs, hiding unsupported field removals while still showing new fields and mapping parameter changes.
  • Remove deleted mapping fields from index templates so future concrete indices do not inherit fields that no longer exist in the schema.
  • Keep settings diffs complete, and report new index templates as created instead of dumping the full config diff.

Behavioral change: index templates now drop removed fields

This is more than an output change. Previously, merge_properties preserved existing template fields forever (a guard added after a near-SEV caused by dropping a template field while deployed indexers still published it). This PR removes that guard: templates are now written to exactly match the schema, so removed fields are dropped and stale fields no longer accumulate.

The operational caveat (now documented in ForIndexTemplate#put_index_template): new rollover indices are auto-created from the template at indexing time with dynamic: strict mappings, so a field must only be removed from the schema once no deployed indexer still publishes it (and no old events containing it will be replayed). Dropped fields appear as deletions in the reported template diff, giving operators a chance to catch a premature removal. Concrete indices are unaffected — their existing fields are always preserved, since the datastore doesn't support removing index mapping fields.

Context

Mapping removals are unsupported by Elasticsearch/OpenSearch concrete indices, so showing removed fields every admin run just makes the diffs harder to read. Index templates can be rewritten without those fields, so this keeps existing concrete indices stable while allowing future indices to reflect the current schema. This addresses #766 and the review feedback to improve elasticgraph-admin directly instead of adding an elasticgraph-admin_lambda opt-in.

Verification

  • script/lint
  • script/spellcheck
  • script/type_check
  • script/quick_build
  • Full CI (including datastore-backed integration/acceptance specs) is green.

@jwils jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from 7123aca to 37b0486 Compare May 22, 2026 23:02
myronmarston

This comment was marked as resolved.

@jwils jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from 37b0486 to c6e3e1f Compare May 22, 2026 23:43
@jwils jwils changed the title Hide delete changes in admin lambda output Improve admin mapping change output May 22, 2026
@jwils jwils marked this pull request as ready for review May 22, 2026 23:49
@jwils jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from c6e3e1f to a9a165f Compare May 23, 2026 00:06
@jwils jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from a9a165f to c72908c Compare June 16, 2026 22:54
@jwils jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from c72908c to a335f6c Compare June 17, 2026 17:40
@jwils jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from 2e5fc2b to fcba5e2 Compare July 1, 2026 18:15
Dropping a field from an index template is a real datastore-state change
with operational risk (new rollover indices are auto-created from the
template with dynamic: strict mappings), and we previously had a near-SEV
from dropping a template field while old indexers still published it.
Re-record that history and the required schema evolution workflow at the
decision point, since this PR removes the merge_properties guard that
previously preserved template fields.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants